Display device

ABSTRACT

According to an aspect, a display device includes a display unit and a control unit. The display unit stereoscopically displays a display object. When a movement of an object is detected in a three-dimensional space where the display object is stereoscopically displayed, the control unit for changes the display object in the three-dimensional space according to the movement of the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Japanese Application No.2011-143070, filed on Jun. 28, 2011, the content of which isincorporated by reference herein in its entirety.

1. Technical Field

The present disclosure relates to a display device.

2. Description of the Related Art

Some display devices such as mobile phones with a display unit canstereoscopically display an image and so on (see e.g., Japanese PatentApplication Laid-open No. 2011-95547). The three-dimensional display isimplemented by using binocular disparity.

The three-dimensional display is a user-friendly display manner;however, it has been used just for viewing purposes, and has not beenused for improving the convenience of operations.

For the foregoing reasons, there is a need for a display device that canprovide the user with convenient operations using the three-dimensionaldisplay.

SUMMARY

According to an aspect, a display device includes a display unit, adetecting unit, and a control unit. The display unit stereoscopicallydisplays a display object. The detecting unit detects an object in athree-dimensional space where the display object is stereoscopicallydisplayed. When a movement of the object is detected in thethree-dimensional space, the control unit changes the display objectaccording to the movement of the object.

According to another aspect, a display device includes a display unitand a control unit. The display unit stereoscopically displays a displayobject. When a movement of an object is detected in a three-dimensionalspace where the display object is stereoscopically displayed, thecontrol unit for changes the display object in the three-dimensionalspace according to the movement of the object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front view of a mobile phone according to a firstembodiment;

FIG. 2 is a block diagram of the mobile phone according to the firstembodiment;

FIG. 3 is a diagram for explaining how to detect an operation of pushinga three-dimensional object and how to change the three-dimensionalobject according to the detected operation in the first embodiment;

FIG. 4 is a diagram for explaining how to detect an operation of pushingthe three-dimensional object and how to change the three-dimensionalobject according to the detected operation in the first embodiment;

FIG. 5 is a diagram illustrating an example of information stored inobject data;

FIG. 6 is a diagram illustrating an example of information stored inaction data;

FIG. 7 is a diagram illustrating an example of the information stored inthe action data;

FIG. 8 is a diagram illustrating an example of the information stored inthe action data;

FIG. 9 is a diagram illustrating an example of the information stored inthe action data;

FIG. 10 is a diagram illustrating an example of the information storedin the action data;

FIG. 11 is a diagram illustrating an example of the information storedin the action data;

FIG. 12 is a flowchart of a procedure of a contact detecting process;

FIG. 13 is a flowchart of a procedure of an operation detecting process;

FIG. 14 is a diagram for explaining how to detect an operation ofpushing a three-dimensional object and how to change thethree-dimensional object according to the detected operation in a secondembodiment;

FIG. 15 is a flowchart of a procedure of an operation detecting process;

FIG. 16 is a diagram for explaining how to detect an operation ofpushing a three-dimensional object and how to change thethree-dimensional object according to the detected operation in a thirdembodiment;

FIG. 17 is a diagram for explaining how to detect an operation ofpushing the three-dimensional object and how to change thethree-dimensional object according to the detected operation in thethird embodiment;

FIG. 18 is a flowchart of a procedure of an operation detecting process;

FIG. 19 is a front view of a mobile phone according to a fourthembodiment;

FIG. 20 is a block diagram of the mobile phone according to the fourthembodiment;

FIG. 21 is a diagram for explaining how to detect an operation performedfor a three-dimensional object in the fourth embodiment; and

FIG. 22 is a diagram of a modified example of the mobile phone accordingto the fourth embodiment.

DETAILED DESCRIPTION

Exemplary embodiments of the present invention will be explained indetail below with reference to the accompanying drawings. It should benoted that the present invention is not limited by the followingexplanation. In addition, this disclosure encompasses not only thecomponents specifically described in the explanation below, but alsothose which would be apparent to persons ordinarily skilled in the art,upon reading this disclosure, as being interchangeable with orequivalent to the specifically described components.

In the following description, a mobile phone is used to explain as anexample of the display device; however, the present invention is notlimited to mobile phones. Therefore, the present invention can beapplied to a variety of devices, including but not limited to personalhandyphone systems (PHS), personal digital assistants (FDA), portablenavigation units, personal computers (including but not limited totablet computers, netbooks etc.), media players, portable electronicreading devices, and gaming devices. The present invention can also beapplied to stationary electronic devices that have a plurality ofdisplay units.

First of all, the configuration of a mobile phone (display device) 1according to a first embodiment will be explained below with referenceto FIG. 1 and FIG. 2. FIG. 1 is a front view of the mobile phone 1. FIG.2 is a block diagram of the mobile phone 1.

As illustrated in FIG. 1 and FIG. 2, the mobile phone 1 includes anoperating unit 13, a microphone 15, a receiver 16, a control unit 22, astorage unit 24, a communication unit 26, a sound processor 30, a touchpanel 32, an imaging unit 40, and an imaging unit 42. The operating unit13, the microphone 15, the receiver 16, the touch panel 32, and theimaging unit 40 are exposed to the front surface of the mobile phone 1.

The operating unit 13 has physical button, and outputs a signalcorresponding to a pressed button to the control unit 22. In the exampleillustrated in FIG. 1, the operating unit 13 has only one button;however, the operating unit 13 may have a plurality of buttons.

The microphone 15 acquires an external sound. The receiver 16 outputs avoice of a call partner during a phone call. The sound processor 30converts the sound input from the microphone 15 to a digital signal andoutputs the digital signal to the control unit 22. The sound processor30 also decodes a digital signal input from the control unit 22 andoutputs the decoded signal to the receiver 16.

The communication unit 26 includes an antenna 26 a, and establishes awireless signal path using a code-division multiple access (CDMA)system, or any other wireless communication protocols, with a basestation via a channel allocated by the base station, and performstelephone communication and information communication with the basestation. Any other wired or wireless communication or networkinterfaces, e.g., LAN, Bluetooth, Wi-Fi, NFC (Near Field Communication)may also be included in lieu of or in addition to the communication unit26.

The touch panel 32 displays various pieces of information such ascharacters, graphics, and images, and detects an input operationperformed on a display area such as icon, button, and character inputarea. The touch panel 32 is structured with a display unit 32 a and atouch sensor 32 b so as to overlap each other.

The display unit 32 a includes a display device such as a liquid crystaldisplay (LCD) or an organic electro-luminescence display (OELD), anddisplays various pieces of information according to a control signalinput from the control unit 22. The touch sensor 32 b detects an inputoperation performed on the surface of the touch panel 32, and outputs asignal corresponding to the detected input operation to the control unit22. The detection method in which the touch sensor 32 b detects variousoperations may be any detection method, such as a capacitive typedetection method, a resistive type detection method, and a pressuresensitive type detection method.

The touch panel 32 can display a three-dimensional object. A“three-dimensional object” is a display object such as an image and ashape created so as to look as if the display object isthree-dimensional using disparity. The method of displaying thethree-dimensional object may be a method of realizing a stereoscopicvision using a tool such as glasses, or may be a method of realizing astereoscopic vision with the naked eye.

The imaging units 40 and 42 electronically photograph an image using animage sensor such as a charge-coupled device (CCD) image sensor or acomplementary metal oxide semiconductor (CMOS) image sensor. Each of theimaging units 40 and 42 converts a photographed image to a signal andoutputs the signal to the control unit 22. The imaging units 40 and 42also function as a detector that detects an object for selecting andoperating a three-dimensional object in a space in which thethree-dimensional object is stereoscopically displayed (hereinafter,also referred to “three-dimensional space”, “stereoscopic vision space”or “visual space”).

The imaging units 40 and 42 are configured to set a field angle andlayout so that, even if an object such as a finger is located in anypart of the three-dimensional space, the object can be photographed. Theimaging units 40 and 42 may be a device that acquires an image ofvisible light or may be a device that acquires an image of invisiblelight such as infrared rays.

The control unit 22 includes a central processing unit (CPU) being aprocessing unit and a memory being a storage unit, and implementsvarious functions by executing programs using these hardware resources.Specifically, the control unit 22 reads a program or data stored in thestorage unit 24 to load it to the memory, and causes the CPU to executeinstructions contained in the program loaded to the memory. The controlunit 22 performs read/write of data from/to the memory and the storageunit 24, and controls operations of the communication unit 26, thedisplay unit 32 a, and the like according to the execution result of theinstructions executed by the CPU. When the CPU executes instructions,the data loaded to the memory and the signal input from the touch sensor32 b or so are used as part of parameters and determination conditions.

The storage unit 24 includes one or more non-transitory storage medium,for example, a nonvolatile memory (such as ROM, EPROM, flash card etc.)and/or a storage device (such as magnetic storage device, opticalstorage device, solid-state storage device etc.), and stores thereinvarious programs and data. Examples of the program stored in the storageunit 24 include a control program 24 a. Examples of the data stored inthe storage unit 24 include object data 24 b and action data 24 c. Thestorage unit 24 may include a combination of a portable storage mediumsuch as a memory card and a reader/writer for reading/writing datafrom/to the storage medium. In this case, the control program 24 a, theobject data 24 b, and the action data 24 c may be stored in the storagemedium. The control program 24 a, the object data 24 b, and the actiondata 24 c may be acquired from any other device such as a server throughcommunication by the communication unit 26.

The control program 24 a provides functions for various controls tooperate the mobile phone 1. The function provided by the control program24 a includes a function for controlling a display of athree-dimensional object on the touch panel 32 and a function fordetecting a user's operation performed for the three-dimensional objectdisplayed by the touch panel 32.

The object data 24 b contains information for shapes and characteristicsof a three-dimensional object. The object data 24 b is used to displaythe three-dimensional object. The action data 24 c contains informationfor how an operation performed for a displayed three-dimensional objectacts for the three-dimensional object. When the operation performed forthe displayed three-dimensional object is detected, the action data 24 cis used to change the three-dimensional object. The change mentionedhere includes movement, rotation, deformation, deletion, and so on.

Then, detection of an operation for pushing a three-dimensional objectand a change of the three-dimensional object according to the detectedoperation will be explained below with reference to FIG. 3 and FIG. 4.FIG. 3 and FIG. 4 are diagrams for explaining how to detect an operationof pushing the three-dimensional object and how to change thethree-dimensional object according to the detected operation. At StepS11 illustrated in FIG. 3, the touch panel 32 stereoscopically displaysa three-dimensional object OB1 in a three-dimensional space 50. Thethree-dimensional object OB1 is, for example, an object resembling aball. At Step S11, the touch panel 32 also displays a bottom surface B1that supports the three-dimensional object OB1.

At Step S12, the user places a finger F1 on a location where it is incontact with the three-dimensional object OB1, and keeps the finger F1still as it is. When an actual object is detected in thethree-dimensional space and a state in which the object keeps in contactwith the three-dimensional object OB1 continues for a longer period oftime than a predetermined time, the mobile phone 1 determines that thethree-dimensional object OB1 is selected as an operation target. Themobile phone 1 changes a display mode of the three-dimensional objectOB1, or so, to notify the user that the three-dimensional object OB1 isselected as the operation target.

The determination as to whether the object is in contact with thethree-dimensional object OB1 is performed based on an actual position ofthe object in the three-dimensional space, a shape of thethree-dimensional object OB1, and a calculated position of thethree-dimensional object OB1 in the three-dimensional space. The shapeof the three-dimensional object OB1 is defined in the object data 24 b.

The actual position of the object is calculated based on imagesphotographed by the imaging units 40 and 42. The actual position of theobject may be calculated based on the size of the previously registeredobject, the sizes of the object in the images, and the positions of theobject in the images. The actual position of the object may also becalculated by comparing the size and the position of the object in theimage photographed by the imaging unit 40 with the size and the positionof the object in the image photographed by the imaging unit 42. Thedetection of the object such as the finger may be implemented using aknown technology. When the object is the finger, the process may beperformed by setting a position of the tip of the finger as a positionof the object.

A calculated position of the three-dimensional object OB1 in thethree-dimensional space is calculated based on a position of thethree-dimensional object OB1 on the display surface of the touch panel32 and an amount of “floating” of the three-dimensional object OB1 inthe three-dimensional space. The amount of floating of thethree-dimensional object OB1 in the three-dimensional space may be avalue determined upon display, or may be a value calculated from adifference between positions of the three-dimensional object OB1 in animage for a right eye and in an image for a left eye, which are used tostereoscopically display the three-dimensional object OB1.

The notification indicating that it is selected as the operation targetis implemented by, for example, changing the whole color of thethree-dimensional object OB1 or changing a color near a location, withinthe surface of the three-dimensional object OB1, where thethree-dimensional object OB1 is in contact with the object. Instead ofor in addition to such visual notification, a sound and/or a vibrationmay be used to perform the notification.

In this way, when the state where the real object such as the finger isin contact with the three-dimensional object OB1 is continuouslydetected for a longer period of time than a predetermined time, themobile phone 1 determines that the three-dimensional object OB1 isselected as the operation target. By adding the continuous detection ofthe contact state for a longer period of time than the predeterminedtime to the condition, an unintended three-dimensional object can beprevented from being selected as an operation target during the processof moving the finger in order to operate any other three-dimensionalobject.

It is assumed that after the selection of the three-dimensional objectOB1 as the operation target, as illustrated at Step S13, the user causesthe finger F1 to enter the inside of the three-dimensional object OB1 asif he/she pushes the three-dimensional object OB1. When the operation ofcausing the object to enter the inside of the three-dimensional objectOB1 selected as the operation target is detected, the mobile phone 1changes the three-dimensional object OB1 according to the operation. Howto change the three-dimensional object OB1 is determined based on thetype of the three-dimensional object OB1 defined in the object data 24 band the rule of the change defined in the action data 24 c inassociation with the type.

For example, it is assumed that it is defined in the object data 24 bthat the three-dimensional object OB1 is an elastic body and it is alsodefined in the action data 24 c that if the elastic body is pushed, thenit is deformed in its pushed direction according to the pushed amount.In this case, as illustrated at S14, the mobile phone 1 changes thethree-dimensional object OB1 so that a portion which the finger F1 hasentered is pushed to dent.

Alternatively, it is assumed that it is defined in the object data 24 bthat the three-dimensional object OB1 is a rigid body and it is alsodefined in the action data 24 c that if the rigid body is pushed, thenit is moved in its pushed direction according to the pushed amount. Inthis case, as illustrated at Step S15 in FIG. 4, the mobile phone 1moves the three-dimensional object OB1 in the direction of forwardmovement as if it is pushed away by the finger 151. At Step S15 in FIG.4, the three-dimensional object OB1 is supported by the bottom surfaceB1, so that the three-dimensional object OB1 moves according to acomponent of the force applied by the object, that is, a component thatacts in a direction parallel to the bottom surface B1.

In this way, when the operation of pushing the three-dimensional objectOB1 is detected, the three-dimensional object OB1 is changed based onthe object data 24 b and the action data 24 c, which enables thethree-dimensional object OB1 to be variously changed according to eachoperation. The pushing operation is an operation used in various scenesin the real world, and therefore by detecting an operation of pushingthe three-dimensional object OB1 and executing the correspondingprocess, intuitive and user-friendly operability can be achieved.

The object used to operate the three-dimensional object is not limitedto the finger, and therefore may be a hand, a foot, a stick, a tool, orso. A way to change the three-dimensional object OB1 according to thepushing operation may follow actual physical law or may be that which isactually impossible.

When the pushed direction of the three-dimensional, object is notparallel to the display surface of the touch panel 32, that is, when themoving direction of the detected object intersects the display surfaceof the touch panel 32 or intersects a horizontal plane parallel to thedisplay surface thereof, the mobile phone 1 changes thethree-dimensional object according to the operation. In this way, bystereoscopically determining the operation of pushing thethree-dimensional object, various operations can be performed for thethree-dimensional object. To stereoscopically determine the operation ofpushing the three-dimensional object, a plurality of imaging units aredesirably prepared to photograph the finger F1 or so from differentdirections so that an obstacle will not cause a blind spot.

The object data 24 b and the action data 24 c illustrated in FIG. 2 willbe explained in more detail below with reference to FIG. 5 to FIG. 11.FIG. 5 is a diagram illustrating an example of information stored in theobject data 24 b. FIG. 6 to FIG. 11 are diagrams illustrating examplesof information stored in the action data 24 c.

As illustrated in FIG. 5, the object data 24 b stores thereininformation including type, shape information, color, transparency, andso on for each three-dimensional object. The type indicates physicalcharacteristics of each three-dimensional object. The type isrepresented by a value such as “Rigid body” and “Elastic body”. Theshape information is information indicating the shape of eachthree-dimensional object. The shape information is, for example, a setof vertex coordinates of faces that form the three-dimensional object.The color is surface color of each three-dimensional object. Thetransparency is a degree in which each three-dimensional objecttransmits light. The object data 24 b can hold information for aplurality of three-dimensional objects.

The action data 24 c stores therein information for changes made whenthe pushing operation is detected, for each type of three-dimensionalobjects. As illustrated in FIG. 6, when the type of thethree-dimensional object is “Rigid body”, a change made when the pushingoperation is detected differs according to existence of a fulcrum,existence of an obstacle in its pushed direction, and a pushed speed.The obstacle mentioned here indicates some other three-dimensionalobject. Whether the pushed speed is high or low is determined based on athreshold.

When there is no fulcrum in the three-dimensional object and there is noobstacle in its pushed direction, the three-dimensional object isdisplayed so as to move in its pushed direction according to a pushedamount. Examples of the three-dimensional object displayed in thismanner include blocks, a pen, a book, etc. As for the way to move,whether the three-dimensional object is slid or rotated may bedetermined based on the shape thereof. Whether the three-dimensionalobject is moved together with a pushing object or is moved separatelyfrom a pushing object as if it is flicked by the pushing object may bedetermined based on the pushed speed, or may be determined based on acalculated value or a set value of frictional resistance between thethree-dimensional object and the bottom surface.

When there is no fulcrum in the three-dimensional object and there is afixed obstacle in its pushed direction, then the three-dimensionalobject is displayed so as to move in its pushed direction according tothe pushed amount and stop the movement when it comes in contact withthe obstacle. Examples of the three-dimensional object displayed in thismanner include blocks, a pen, a book, etc. When the pushed speed ishigh, the three-dimensional object may break the obstacle and continueto move. When the three-dimensional object comes in contact with anobstacle while being moved separately from a pushing object as if it isflicked by the pushing object, the three-dimensional object may be movedin an opposite direction as if it has bounced off the obstacle.

When there is no fulcrum in the three-dimensional object, there is anyother rigid body which is not fixed in its pushed direction, and thepushed speed is low, then the three-dimensional object is displayed soas to move in its pushed direction according to the pushed amount andmove together with the any other rigid body after the three-dimensionalobject comes in contact with the any other rigid body. When there is nofulcrum in the three-dimensional object, there is any other rigid bodywhich is not fixed in its pushed direction, and the pushed speed ishigh, then the three-dimensional object is displayed so as to move inthe pushed direction according to the pushed amount. After thethree-dimensional object comes in contact with the any other rigid body,the rigid body is displayed so as to move as if it is flicked by thethree-dimensional object. After coming in contact with any other rigidbody, the three-dimensional object may be stopped on that spot, or maycontinue to move with the speed being slowed down. Examples of acombination of the three-dimensional object and the any other rigid bodydisplayed in this manner include a combination of a ball and a pin ofbowling, a combination of marbles, etc.

When there is no fulcrum in the three-dimensional object, there is anyother rigid body which is not fixed in its pushed direction, and thethree-dimensional object can pass through the any other rigid body, thenthe three-dimensional object is displayed so as to move in its pusheddirection according to the pushed amount and pass through the any otherrigid body even after the three-dimensional object comes in contact withthe any other rigid body, to continuously move as it is. In reality, arigid body is impossible to pass through any other rigid body; however,by enabling such a pass, fresh experiment can be provided to the user.Examples of a combination of the three-dimensional object and the anyother rigid body displayed in this manner include a combination of aball and a pin of bowling, a combination of marbles, etc. It may be setthat the three-dimensional object does not pass through any other rigidbody when a threshold is provided with respect to the pushed speed andthe pushed speed is slower than the threshold.

When there is a fulcrum in the three-dimensional object, thethree-dimensional object is displayed so as to rotate around the fulcrumaccording to its pushed direction and amount. The rotation mentionedhere may be continuous rotation through 360 degrees or may be areciprocating swing motion within a predetermined turning range.Examples of the three-dimensional object displayed in this mannerinclude a pendulum, a sand bag for boxing, a windmill, etc.

As illustrated in FIG. 7, when the type of the three-dimensional objectis “Elastic body”, a change made when a pushing operation is detecteddiffers according to a material, existence of limitation to its changeamount, and a pushed speed. The material mentioned here is an assumedmaterial of the three-dimensional object, which is defined in the objectdata 24 b.

When the material of the three-dimensional object is a rubber-basedmaterial without limitation to its change amount and the pushed speed islow, the three-dimensional object is displayed so as to deform in thepushed direction according to the pushed amount and return to itsoriginal shape when it is released from the pushed state. When thematerial of the three-dimensional object is a rubber-based materialwithout limitation to its change amount and the pushed speed is high,the three-dimensional object is displayed so as to deform in its pusheddirection according to the pushed amount. Thereafter thethree-dimensional object is displayed so as to move in the pusheddirection as if it is flicked away while returning to its originalshape. Examples of the three-dimensional object displayed in this mannerinclude a rubber ball, a rubber eraser, etc.

When the material of the three-dimensional object is a rubber-basedmaterial with limitation to the change amount, the three-dimensionalobject is displayed so as to deform in its pushed direction within adeformable range according to the pushed amount. Then, when thesubsequent pushing operation is detected, the three-dimensional objectis displayed so as to move in the pushed direction while returning toits original shape. Examples of the three-dimensional object displayedin this manner include a rubber ball, a rubber eraser, etc.

When the material of the three-dimensional object is a metal-basedmaterial, the three-dimensional object is displayed so that it isdeformed in its pushed direction within a deformable range according tothe pushed amount. Then, when it is released from the pushed state, thethree-dimensional object is displayed so that returning to its originalshape and deformation are repeated (so as to vibrate). If thethree-dimensional object is pushed in any direction other than thedeformable direction, the three-dimensional object moves similarly tothe rigid body. Examples of the three-dimensional object displayed inthis manner include a plate spring, a helical spring, etc.

As illustrated in FIG. 8, when the type of the three-dimensional objectis “Plastic body”, the three-dimensional object is displayed so that itspushed portion dents and the entire shape is thereby deformed. Examplesof the three-dimensional object displayed in this manner include clay,etc.

As illustrated in FIG. 9, when the type of the three-dimensional objectis “Liquid”, a change made when the pushing operation is detecteddiffers according to the pushed speed. When the pushed speed is low, apushing object is displayed so as to be soaked in the three-dimensionalobject, that is, in the liquid. When the pushed speed is medium, thepushing object is displayed so that it is soaked in the liquid withripples spreading across the liquid. When the pushed speed is high, thepushing object is displayed so that it is soaked in the liquid withwater splashes from the liquid. Examples of the three-dimensional objectdisplayed in this manner include water in a cup, etc.

As illustrated in FIG. 10, when the type of the three-dimensional objectis “Gas”, a change made when the pushing operation is detected differsaccording to the pushed speed. When the pushed speed is low, thethree-dimensional object, that is, the gas is displayed so that it isblocked by the pushing object to float around the periphery thereof.When the pushed speed is medium, the gas is displayed so as to bescattered by the pushing object. When the pushed speed is high, the gasis displayed so that eddy is produced in the gas due to turbulent flowin the back side of the moving direction of the pushing object. Examplesof the three-dimensional object displayed in this manner include smoke,etc.

As illustrated in FIG. 11, when the type of the three-dimensional objectis “Aggregate”, a change made when the pushing operation is detecteddiffers according to how elements of the aggregate are combined. Whenthe elements of the aggregate are not combined, the three-dimensionalobject is displayed so that its pushed portion dents and the entireshape as the aggregate is thereby changed. Examples of thethree-dimensional object displayed in this manner include sands, sugar,etc.

When the elements of the aggregate are combined, the three-dimensionalobject is displayed so that its pushed portion dents and the entireshape as the aggregate is thereby changed. Moreover, it may be displayedso that any elements other than the pushed portion move by being pulledby the element of the pushed portion. Examples of the three-dimensionalobject displayed in this manner include a chain, etc.

When the elements of the aggregate are not combined but attractive forceor repulsive force acts between the elements and a pushing object, thethree-dimensional object is displayed so as to move without contactingthe pushing object. When the attractive force acts between the elementsand the pushing object, the three-dimensional object is attracted to thepushing object without contacting the pushing object when it enterswithin a predetermined distance to the pushing object. When therepulsive force acts between the elements and the pushing object, thethree-dimensional object is repelled from the pushing object withoutcontacting the pushing object when it enters within a predetermineddistance to the pushing object. Examples of a combination of thethree-dimensional object and the pushing object displayed in this mannerinclude a combination of iron powder and a magnet, etc.

In this manner, by changing the three-dimensional object based on theinformation stored in the object data 24 b and the information stored inthe action data 24 c, the three-dimensional object can be variouslychanged according to the pushing operation. The information stored inthe object data 24 b and in the action data 24 c is not limited to theexample, and therefore may be appropriately varied depending on intendeduse or so. For example, settings may be made so that the way to changethe three-dimensional object is switched according to the type and thesize of the pushing object and/or the size of a contact area between thepushing object and the three-dimensional object.

Then, a procedure performed by the mobile phone 1 related to theoperation of pushing the three-dimensional object will be explained withreference to FIG. 12 and FIG. 13. FIG. 12 is a flowchart of a procedureof the contact detecting process of the three-dimensional object. Theprocedure illustrated in FIG. 12 is implemented by the control unit 22executing the control program 24 a triggered by detection of apredetermined operation or the like.

As illustrated in FIG. 12, first of all, at Step S101, the control unit22 stereoscopically displays a three-dimensional object based on theobject data 24 b. The object data 24 b may be previously stored in thestorage unit 24 or may be acquired from any other device such as aserver through communication by the communication unit 26.

Subsequently, at Step S102, the control unit 22 determines whetherdetecting units, that is, the imaging units 40 and 42 have detected apredetermined object. The predetermined object is, for example, a user'sfinger. When the predetermined object has not been detected (No at StepS102), then at Step S108, the control unit 22 determines whetheroperation completion has been detected.

For example, the operation completion may be detected when apredetermined operation is performed for the operating unit 13, or maybe detected when a predetermined operation is performed for the touchpanel 32. The operation completion may also be detected when apredetermined user's hand gesture is photographed by at least one of theimaging units 40 and 42. When the operation completion has been detected(Yes at Step S108), the control unit 22 ends the contact detectingprocess. When the operation completion has not been detected (No at StepS108), the control unit 22 re-executes Step S102 and the subsequentsteps.

When the predetermined object has been detected (Yes at Step S102), thenat Step S103, the control unit 22 determines the type of thepredetermined object. The type of the predetermined object is determinedbased on the size, the shape, the color, and so on of the object inimages photographed by the imaging units 40 and 42. Subsequently, atStep S104, the control unit 22 looks for a three-dimensional object incontact with the predetermined object.

When there is no three-dimensional object in contact with thepredetermined object (No at Step S105), then at Step S108, the controlunit 22 determines whether the operation completion has been detected.When the operation completion has been detected (Yes at Step S108), thecontrol unit 22 ends the contact detecting process. When the operationcompletion has not been detected (No at Step S108), the control unit 22re-executes Step S102 and the subsequent steps.

When the three-dimensional object in contact with the predeterminedobject has been found (Yes at Step S105), then at Step S106, the controlunit 22 determines the type of the three-dimensional object in contactwith the predetermined object based on the object data 24 b. Then atStep S107, the control unit 22 executes an operation detecting processexplained later. Thereafter, at Step S108, the control unit 22determines whether the operation completion has been detected. When theoperation completion has been detected (Yes at Step S108), the controlunit 22 ends the contact detecting process. When the operationcompletion has not been detected (No at Step S108), the control unit 22re-executes Step S102 and the subsequent steps.

FIG. 13 is a flowchart of a procedure of the operation detectingprocess. The procedure illustrated in FIG. 13 is implemented by thecontrol unit 22 executing the control program 24 a.

As illustrated in FIG. 13, first of all, at Step S201, the control unit22 acquires a contact time for which the predetermined object keeps incontact with the three-dimensional object. Then at Step S202, thecontrol unit 22 determines whether the predetermined object has beenmoved to the inside of the three-dimensional object. When thepredetermined object has not been moved to the inside of thethree-dimensional object (No at Step S202), the control unit 22re-executes Step S201 and the subsequent step.

When the predetermined object has been moved to the inside of thethree-dimensional object (Yes at Step S202), then at Step S203, thecontrol unit 22 determines whether the contact time is longer than apredetermined time. When the contact time is not longer than thepredetermined time (No at Step S203), it is determined that thethree-dimensional object is not the operation target, and therefore thecontrol unit 22 ends the operation detecting process.

When the contact time is longer than the predetermined time (Yes at StepS203), then at Step S204, the control unit 22 calculates a speed of thepredetermined object. At Step S205, the control unit changes thethree-dimensional object based on the type, the location, and the speedof the predetermined object and based on the type and the like of thethree-dimensional object. A specific way to change the three-dimensionalobject is determined according to the action data 24 c.

Subsequently, the control unit 22 determines at Step S206 whether thepredetermined object has been moved to the outside of thethree-dimensional object. When the predetermined object has not beenmoved to the outside of the three-dimensional object, that is, when thepushing operation is continued (No at Step S206), the control unit 22re-executes Step S204 and the subsequent steps.

When the predetermined object has been moved to the outside of thethree-dimensional object, that is when the three-dimensional object hasbeen released (Yes at Step S206), then at Step S207, the control unit 22determines whether the change of the three-dimensional object iscontinued. For example, when it is defined in the action data 24 c thatthe vibration is continued for a predetermined time even after therelease, it is determined that the change of the three-dimensionalobject is continued.

When the change of the three-dimensional object is continued (Yes atStep S207), then at Step S208, the control unit 22 changes thethree-dimensional object, and thereafter re-executes Step S207 and thesubsequent step. When the change of the three-dimensional object is notcontinued (No at Step S207), the control unit 22 ends the operationdetecting process.

As explained above, the first embodiment is configured to variouslychange the three-dimensional object according to the pushing operation,thus providing a user-friendly operation method to users.

A second embodiment will be explained below. The mobile phone 1according to the second embodiment is different in a procedure of theoperation detecting process executed based on the functions provided bythe control program 24 a from that according to the first embodiment.However, in terms of the hardware, the mobile phone 1 according to thesecond embodiment is configured in the same manner as that of the mobilephone 1 according to the first embodiment. Therefore, in the secondembodiment, explanation that overlaps with the explanation in the firstembodiment is omitted, and the operation detecting process will bemainly explained below.

First of all, detection of an operation of pushing a three-dimensionalobject and a change of the three-dimensional object according to thedetected operation will be explained with reference to FIG. 14. FIG. 14is a diagram for explaining how to detect an operation of pushing thethree-dimensional object and how to change the three-dimensional objectaccording to the detected operation. At Step S21 illustrated in FIG. 14,the user brings the finger F1 into contact with the three-dimensionalobject, and at Step S22, the user causes the finger F1 to enter theinside of the three-dimensional object OB1.

When an actual object has been detected in the three-dimensional spaceand the state in which the object moves to the inside of thethree-dimensional object after coming in contact with thethree-dimensional object OB1 is continued for a longer period of timethan the predetermined time, then the mobile phone 1 determines that thethree-dimensional object OB1 has been selected as the operation target.The mobile phone 1 changes the display mode of the three-dimensionalobject OB1, or so, to notify the user that the three-dimensional objectOB1 has been selected as the operation target. Moreover, as illustratedat Step S23, the mobile phone 1 changes the three-dimensional object OB1according to the operation by the finger F1 after the detection of thecontact as if it is already selected as an object of the pushingoperation at the stage of Step S21.

In this way, it is configured that the pushing operation can be detectedeven if the object does not remain in that spot after a contact betweenthe object and the three-dimensional object is detected. Therefore, theuser can quickly start the operation of pushing the three-dimensionalobject. By adding whether the state in which the object after thecontact moves to the inside of the three-dimensional object OB1continues for a longer period of time than the predetermined time to thecondition, an unintended three-dimensional object can be prevented frombeing selected as an operation target during the process of moving thefinger in order to operate any other three-dimensional object.

Then, a procedure of the operation detecting process according to thesecond embodiment will be explained below with reference to FIG. 15.FIG. 15 is a flowchart of the procedure of the operation detectingprocess. The procedure illustrated in FIG. 15 is implemented by thecontrol unit 22 executing the control program 24 a. It should be notedthat the procedure of the contact detecting process is the same as thatillustrated in FIG. 12.

As illustrated in FIG. 15, first of all, at Step S301, the control unit22 determines whether the predetermined object has been moved to theinside of the three-dimensional object. When the predetermined objecthas not been moved to the inside of the three-dimensional object (No atStep S301), it is determined that the three-dimensional object is notthe operation target, and therefore the control unit 22 ends theoperation detecting process.

When the predetermined object has been moved to the inside of thethree-dimensional object (Yes at Step S301), then at Step S302, thecontrol unit 22 determines whether an elapsed time since the detectionof the contact is longer than a predetermined time. When the elapsedtime is not longer than the predetermined time (No at Step S302), thecontrol unit 22 re-executes Step S301 and the subsequent step.

When the elapsed time is longer than the predetermined time (Yes at StepS302), then at Step S303, the control unit 22 calculates a speed of thepredetermined object. At Step S304, the control unit changes thethree-dimensional object based on the type, the location, and the speedof the predetermined object and based on the type and the like of thethree-dimensional object. A specific way to change the three-dimensionalobject is determined according to the action data 24 c.

Subsequently, the control unit 22 determines at Step S305 whether thepredetermined object has been moved to the outside of thethree-dimensional object. When the predetermined object has not beenmoved to the outside of the three-dimensional object, that is, when thepushing operation is continued (No at Step S305), the control unit 22re-executes Step S303 and the subsequent steps.

When the predetermined object has been moved to the outside of thethree-dimensional object, that is, when the three-dimensional object hasbeen released (Yes at Step S305), then at Step S306, the control unit 22determines whether the change of the three-dimensional object iscontinued. For example, when it is defined in the action data 24 c thatthe vibration is continued for a predetermined time even after therelease, it is determined that the change of the three-dimensionalobject is continued.

When the change of the three-dimensional object is continued (Yes atStep S306), then at Step S307, the control unit 22 changes thethree-dimensional object, and thereafter re-executes Step S306 and thesubsequent step. When the change of the three-dimensional object is notcontinued (No at Step S306), the control unit 22 ends the operationdetecting process.

As explained above, the second embodiment is configured to recognize thepushing operation even when the state in which the object such as thefinger is in contact with the three-dimensional object does not continuefor a longer period of time than the predetermined time. Therefore, theuser can quickly start the operation of pushing the three-dimensionalobject.

A third embodiment will be explained below. The mobile phone 1 accordingto the third embodiment is different in a procedure of the operationdetecting process executed based on the functions provided by thecontrol program 24 a from that according to the first embodiment.However, in terms of the hardware, the mobile phone 1 according to thethird embodiment is configured in the same manner as that of the mobilephone 1 according to the first embodiment. Therefore, in the thirdembodiment, explanation that overlaps with the explanation in the firstembodiment is omitted, and the operation detecting process will bemainly explained below.

First of all, detection of an operation of pushing a three-dimensionalobject and a change of the three-dimensional object according to thedetected operation will be explained with reference to FIG. 16 and FIG.17. FIG. 16 and FIG. 17 are diagrams for explaining how to detect anoperation of pushing the three-dimensional object and how to change thethree-dimensional object according to the detected operation. At StepS31 illustrated in FIG. 16, the touch panel 32 stereoscopically displaysthe three-dimensional object OB1 in the three-dimensional space. Theuser brings the finger F1 into contact with the three-dimensionalobject.

It is assumed that the user causes the finger F1 to enter the inside ofthe three-dimensional object OB1. When it is detected that the object incontact with the three-dimensional object OB1 has been moved to theinside of the three-dimensional object OB1, then as illustrated at StepS32, the mobile phone 1 changes the three-dimensional object OB1, fromthat point on, according to the operation by the finger F1. In theexample of FIG. 16, at Step S32, the three-dimensional object OB1 startsmoving in synchronization with movement of the finger F1.

As illustrated at Step S33, the mobile phone 1 determines thethree-dimensional object OB1 as the operation target at the stage whenthe movement of the finger F1 to the inside of the three-dimensionalobject OB1 has been continued for a longer period of time than thepredetermined time. The mobile phone 1 changes the display mode of thethree-dimensional object OB1, or so, to notify the user that thethree-dimensional object OB1 is determined as the operation target.Thereafter, the mobile phone 1 also continues changing thethree-dimensional object OB1 while the movement of the finger F1 to theinside of the three-dimensional object OB1 is detected.

As illustrated at Step S34 in FIG. 17, when the movement of the fingerF1 to the inside of the three-dimensional object OB1 becomesundetectable before the predetermined time elapses, the mobile phone 1adds a reverse change to the change added so far, to thethree-dimensional object OB1. Consequently, the three-dimensional objectOB1 is displayed at the same position as that at the stage of Step S31in the same state. The speed of the reverse change added to thethree-dimensional object OB1 may be higher than the speed at which thechange is added to the three-dimensional object OB1 so far. That is, thethree-dimensional object OB1 may be reversely changed as if it isreversely reproduced at a high speed.

In this way, by starting adding the change to the three-dimensionalobject from the stage when the entry of the object to the inside of thethree-dimensional object is detected, the user can recognize that thethree-dimensional object is being selected before the selection isdetermined. As a result, the user is able to know, at an early point,whether an intended three-dimensional object is selected. When anunintended three-dimensional object is selected, the user can return theunintentionally selected three-dimensional object to its original stateby stopping the operation before the predetermined time elapses.

Until the movement of the finger F1 to the inside of thethree-dimensional object OB1 is continued for a longer period of timethan the predetermined time, the three-dimensional object with thechange added thereto may be displayed in a mode (e.g., translucent mode)different from the normal mode or from the mode in which the selectionas the operation target is determined. By changing the display mode inthis manner, the user can easily determine the state of thethree-dimensional object.

Then, a procedure of the operation detecting process according to thethird embodiment will be explained below with reference to FIG. 18. FIG.18 is a flowchart of the procedure of the operation detecting process.The procedure illustrated in FIG. 18 is implemented by the control unit22 executing the control program 24 a. It should be noted that theprocedure of the contact detecting process is the same as thatillustrated in FIG. 12.

As illustrated in FIG. 18, first of all, at Step S401, the control unit22 determines whether the predetermined object has been moved to theinside of the three-dimensional object. When the predetermined objecthas not been moved to the inside of the three-dimensional object (No atStep S401), it is determined that the three-dimensional object is notthe operation target, and therefore the control unit 22 ends theoperation detecting process.

When the predetermined object has been moved to the inside of thethree-dimensional object (Yes at Step S401), then at Step S402, thecontrol unit 22 calculates a speed of the predetermined object. At StepS403, the control unit changes the three-dimensional object based on thetype, the location, and the speed of the predetermined object and basedon the type and the like of the three-dimensional object. A specific wayto change the three-dimensional object is determined according to theaction data 24 c.

Subsequently, at Step S404, the control unit 22 determines whether anelapsed time since the detection of the contact is longer than apredetermined time. When the elapsed time is not longer than thepredetermined time, that is, when the three-dimensional object is notdetermined as an object of the pushing operation (No at Step S404), thenat Step S405, the control unit 22 determines whether the movement of thepredetermined object to the inside direction of the three-dimensionalobject is continued.

When the movement of the predetermined object to the inside direction ofthe three-dimensional object is continued (Yes at Step S405), thecontrol unit 22 re-executes Step S402 and the subsequent steps. When themovement of the predetermined object to the inside direction of thethree-dimensional object is not continued (No at Step S405), then atStep S406, the control unit 22 reversely changes the three-dimensionalobject OB1 to be returned to its original state. The control unit 22then ends the operation detecting process.

When the elapsed time since the detection of the contact is longer thanthe predetermined time (Yes at Step S404), then at Step S407, thecontrol unit 22 determines whether the predetermined object has beenmoved to the outside of the three-dimensional object. When thepredetermined object has not been moved to the outside of thethree-dimensional object, that is, when the pushing operation iscontinued (No at Step S407), the control unit 22 re-executes Step S402and the subsequent steps.

When the predetermined object has been moved to the outside of thethree-dimensional object, that is, when the three-dimensional object hasbeen released (Yes at Step S407), then at Step S408, the control unit 22determines whether the change of the three-dimensional object iscontinued. For example, when it is defined in the action data 24 c thatthe vibration is continued for a predetermined time even after therelease, it is determined that the change of the three-dimensionalobject is continued.

When the change of the three-dimensional object is continued (Yes atStep S408), then at Step S409, the control unit 22 changes thethree-dimensional object, and thereafter re-executes Step S408 and thesubsequent step. When the change of the three-dimensional object is notcontinued (No at Step S408), the control unit 22 ends the operationdetecting process.

As explained above, the third embodiment is configured to change thethree-dimensional object according to the operation from the time whenthe pushing operation is detected. Therefore, the user can therebyeasily recognize the three-dimensional object determined as a target ofthe pushing operation.

A fourth embodiment will be explained below. The embodiments areconfigured to detect the object that operates the three-dimensionalobject based on the images photographed by the imaging units; however,some other detection methods may be used. For example, a capacitive typetouch sensor can detect a position of a finger that does not touch thetouch sensor by increasing its sensitivity. Therefore, an example ofusing the touch sensor as a detector that detects an object operating athree-dimensional object is explained in the fourth embodiment. In thefollowing explanation, the same signs as these of the already explainedcomponents are assigned to the same components as the already explainedcomponents. Explanation that overlaps with the above explanation may beomitted.

First of all, a configuration of a mobile phone (display device) 2according to the fourth embodiment will be explained below withreference to FIG. 19 and FIG. 20. FIG. 19 is a front view of the mobilephone 2. FIG. 20 is a block diagram of the mobile phone 2.

As illustrated in FIG. 19 and FIG. 20, the mobile phone 2 includes theoperating unit 13, the microphone 15, the receiver 16, the control unit22, the storage unit 24, the communication unit 26, the sound processor30, and the touch panel 32.

The touch panel 32 displays various pieces of information such ascharacters, graphics, and images, and detects an input operationperformed on a predetermined display area such as icon, button, andcharacter input area. The touch panel 32 is structured with the displayunit 32 a and the touch sensor 32 b so as to overlap each other. Thetouch sensor 32 b according to the present embodiment is a capacitivetype touch sensor. The touch sensor 32 b functions also as a detectorthat detects fingers operating a three-dimensional object.

Then, the detection of an operation performed for a three-dimensionalobject will be explained with reference to FIG. 21. FIG. 21 is a diagramfor explaining how to detect an operation performed for athree-dimensional object. As illustrated in FIG. 21, the touch panel 32stereoscopically displays the three-dimensional object OB1 in thethree-dimensional space. Also, as illustrated in FIG. 21, the userbrings the finger F1 into contact with the three-dimensional object OB1.

The mobile phone 2 detects a location of the finger F1 using the touchsensor 32 b. The touch sensor 32 b can detect the location of the fingerF1 in an X-axis direction and a Y-axis direction by increasing itssensitivity even if, for example, a distance from the finger F1 to thesurface of the touch panel 32 in a Z-axis direction is about 10 cm. Thetouch sensor 32 b can also detect a distance D2 from the finger F1 tothe surface of the touch panel 32 in the Z-axis direction based on themagnitude of the capacitance.

The mobile phone 2 can detect the contact between the finger F1 and thethree-dimensional object OB1 and can detect the operation of pushing thethree-dimensional object OB1 performed by the finger F1, based on thethus detected location of the finger F1 in the three-dimensional space.

As explained above, the fourth embodiment is configured to use the touchsensor as a detector, so that the operation performed for thethree-dimensional object can be detected even by a display devicewithout the imaging unit.

To detect the operation performed for the three-dimensional object, theimaging unit and the touch sensor may be used in combination with eachother. When the imaging unit is combined with the touch sensor,respective detection results may be averaged to specify the location ofthe finger F1. A weighted average may be used, such that weighting ofthe detection results of the touch sensor is increased in an area nearthe touch panel 32 where the imaging unit 40 is difficult to acquire theimage of the finger F1 and weighting of the detection results of theimaging unit 40 is increased in an area far from the touch panel 32where the detection precision of the touch sensor becomes low.

To prevent the touch sensor from not detecting a location of a fingerwith high precision because any other finger or so may block the finger,a plurality of touch sensors may be used to detect an operationperformed for the three-dimensional object. FIG. 22 is a diagram of aconfiguration example of a mobile phone 3 that uses a plurality of touchsensors to detect an operation performed for the three-dimensionalobject.

The mobile phone 3 includes a first housing 3 a, a second housing 3 b,and a hinge 3 c. The hinge 3 c couples the first housing 3 a and thesecond housing 3 b so as to be openable and closable. The first housing3 a is provided with the touch panel 32 including the touch sensor 32 b,and the second housing 3 b is provided with a touch panel 34 including atouch sensor 34 b. As illustrated in FIG. 22, the touch sensor 32 b andthe touch sensor 34 b contact the three-dimensional space at differentangles when the first housing 3 a and the second housing 3 b are fixedto each other at an angle of about 90 degrees.

The touch sensor 32 b can detect the location of the finger F1 in theX-axis direction and the Y-axis direction. The touch sensor 32 b canalso detect the distance D2 from the finger F1 to the surface of thetouch panel 32 in the Z-axis direction, based on the magnitude of thecapacitance. The touch sensor 34 b can detect the location of the fingerF1 in the X-axis direction and the Z-axis direction. The touch sensor 34b can also detect a distance D4 from the finger F1 to the surface of thetouch panel 34 in the Y-axis direction, based on the magnitude of thecapacitance.

In this manner, by detecting the finger F1 from different directions,even if there is some obstacle, the location of the finger F1 can bedetected from either one of the directions. When the finger F1 isdetected from the different directions, one of the touch panels maydisplay the three-dimensional object, and the other touch panel may stopdisplaying or may two-dimensionally display guidance or so. The touchpanel which does not display the three-dimensional object may beconfigured as a mere touch sensor.

Examples of the application of the display device explained in theembodiments and modified examples thereof will be explained below. Athree-dimensional object (display object) being an operation target maybe any object resembling an object actually existing such as a book,blocks, a spoon, chopsticks, playing cards, clay, and an instrument, ormay be any object not existing such as a virtual avatar, a character ina game, and an augmented reality (AR) tag in virtual reality. The changeadded to the three-dimensional object according to the detectedoperation is not limited to the movement, the deformation, the deletion,or the like. The change added to the three-dimensional object accordingto the pushing operation is not limited to these of the embodiments, andmay therefore be changed according to the type of the three-dimensionalobject.

For example, when a three-dimensional object resembling clay(hereinafter, “clay”) is determined as an operation target, the clay maybe deformed according to the pushing operation, so that the user canform the clay into an arbitrary shape. In addition, the viscosity of theclay may be decreased with the elapse of time as if the clay is gettingdry. When an operation of soaking a finger or hand in water being thethree-dimensional object and pushing the clay with the finger or hand isdetected, the viscosity of the clay may be increased.

When a three-dimensional object resembling a phonograph record(hereinafter, “phonograph record”) is determined as an operation target,it may be set so that the phonograph record is rotated around its centerand a sound is reproduced according to the pushing operation. Bysynchronizing the rotation with the sound reproduction, technique suchas a scratch performed by a disc jockey may be virtually achieved.

The aspects of the present invention represented in the embodiments canbe arbitrarily modified without departing from the spirit of the presentinvention. Moreover, the embodiments may be combined with each other asrequired. For example, the control program 24 a represented in theembodiments may be divided into a plurality of modules or may beintegrated with any other program. In the embodiments, the fingers areused to operate the three-dimensional object; however, a stick-likeobject or so of which end can be charged with static electricity may beused instead of the fingers.

In the embodiments, the example of using the imaging units and the touchsensor are used as a detector in order to detect the three-dimensionalobject; however, the detector is not limited thereto. For example, asensor using a Time-of-Flight (TOF) method may be used instead of theimaging unit. When a plurality of proximity sensor or the like capableof detecting a movement of the three-dimensional object in a planardirection of the three-dimensional space are arranged substantiallyhorizontally with respect to a moving direction of the object,displacement of the object can be detected even in a noncontact manner,and therefore these devices may also be used. It is preferable that thedisplacement of the object can be detected without providing the sensoror the like in the object. If the sensor or the like is not provided inthe object, then this does not have to attach an acceleration sensor tothe finger or does not have to move a display device itself with anacceleration sensor, which leads to cost reduction.

The embodiments have explained the case where the three-dimensionalobject projects toward the user side; however, the present invention isalso applicable to a case of representing the three-dimensional objectas if it is present in a deeper side than the display unit. In thiscase, a sensor and a camera may be provided in the back side of thedisplay unit. When the display device is a mobile phone, many mobilephones are provided with an in-camera for photographing the userhimself/herself and an out-camera for photographing landscapes and thelike. Therefore, it may be configured to capture the displacement of theobject in the back side by using the out-camera.

The embodiments are configured that the display device singly detects anoperation performed for the three-dimensional object; however, thedisplay device may collaborate with a server to detect an operationperformed for the three-dimensional object. In this case, the displaydevice successively transmits information detected by the detector tothe server, and the server detects the operation to notify the displaydevice of the detection result. Such a configuration as above enablesthe load of the display device to be reduced.

The advantages are that one embodiment of the invention provides adisplay device that can provide the user with convenient operations.

What is claimed is:
 1. A display device, comprising: a display unit forstereoscopically displaying a display object in a three-dimensionalspace; a detecting unit for detecting an actual object in athree-dimensional space where the display object is stereoscopicallydisplayed; and a control unit for changing, when a movement of theactual object is detected in the three-dimensional space, a display formof the same display object according to the movement of the actualobject and a type of the display object, wherein the detecting unit isconfigured to detect a state in which the actual object movescontinuously from outside of the display object to inside thereof for alonger period of time than a predetermined time, the control unit isconfigured to start changing the display object from when the movementof the actual object to the inside of the display object is detected,and the control unit is configured to display, when the movement of theactual object to the inside of the display object becomes undetectablebefore the predetermined time elapses and after the display object isstarted to change, a reverse change such that the change of the displayobject is returned to its original state.
 2. The display deviceaccording to claim 1, wherein the control unit is configured to displaythe reverse change in a shorter time than a time taken by the change. 3.The display device according to claim 1, wherein the control unit isconfigured to change the display object by moving or rotating thedisplay object or by combining the movement and the rotation with amovement of the actual object.
 4. The display device according to claim1, wherein the control unit is configured to change the display objectby deforming the display object with a movement of the actual object. 5.The display device according to claim 1, wherein the detecting unitincludes an imaging unit for photographing an image of the actualobject.
 6. The display device according to claim 1, wherein thedetecting unit includes a touch sensor for detecting a capacitance ofthe actual object.
 7. The display device according to claim 1, whereinthe detecting unit includes an imaging unit for photographing an imageof the actual object and a touch sensor for detecting a capacitance ofthe actual object, and the control unit is configured to use a weightedaverage of a detection result via the imaging unit and a detectionresult via the touch sensor to specify the location of the actualobject.
 8. The display device according to claim 1, wherein the type ofthe display object is any one selected from the group consisting of arigid body, an elastic body, a plastic body, liquid, gas, and anaggregate.
 9. The display device according to claim 1, wherein thecontrol unit is configured to change the display object according to aspeed of the actual object in the movement of the actual object.
 10. Thedisplay device according to claim 1, wherein the display unit is furtherconfigured to display a bottom surface in the three-dimensional space,the bottom surface supporting the display object, and the control unitis configured to change the display object while the display object isbeing supported by the bottom surface.